Document transcription system training

ABSTRACT

A system is provided for training an acoustic model for use in speech recognition. In particular, such a system may be used to perform training based on a spoken audio stream and a non-literal transcript of the spoken audio stream. Such a system may identify text in the non-literal transcript which represents concepts having multiple spoken forms. The system may attempt to identify the actual spoken form in the audio stream which produced the corresponding text in the non-literal transcript, and thereby produce a revised transcript which more accurately represents the spoken audio stream. The revised, and more accurate, transcript may be used to train the acoustic model, thereby producing a better acoustic model than that which would be produced using conventional techniques, which perform training based directly on the original non-literal transcript.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is related to a concurrently-filed and commonly-ownedU.S. Patent Application entitled “Automated Extraction of SemanticContent and Generation of a Structured Document from Speech,” which ishereby incorporated by reference.

BACKGROUND

1. Field of the Invention

The present invention relates to document transcription systems, andmore particularly, to techniques for training document transcriptionsystems.

2. Related Art

It is desirable in many contexts to record human speech in a writtendocument. In general, the term “transcription” refers to the process ofrecording speech in a textual document referred to as a “transcript” ofthe speech. In the legal profession, for example, transcriptioniststranscribe testimony given in court proceedings and in depositions toproduce a written transcript of the testimony. Similarly, in the medicalprofession, transcripts are produced of diagnoses, prognoses,prescriptions, and other information dictated by doctors and othermedical professionals. Transcripts in these and other fields typicallyneed to be highly accurate (as measured in terms of the degree ofcorrespondence between the original speech and the resulting transcript)because of the reliance placed on the resulting transcripts and the harmthat could result from an inaccuracy (such as providing an incorrectprescription drug to a patient). High degrees of reliability may,however, be difficult to obtain consistently for a variety of reasons,such as variations in: (1) features of the speakers whose speech istranscribed (e.g., accent, volume, dialect, speed); (2) externalconditions (e.g., background noise); (3) the transcriptionist ortranscription system (e.g., imperfect hearing or audio capturecapabilities, imperfect understanding of language); or (4) therecording/transmission medium (e.g., paper, analog audio tape, analogtelephone network).

At first, transcription was performed solely by human transcriptionistswho would listen to speech, either in real-time (i.e., in person by“taking dictation”) or by listening to a recording. One benefit of humantranscriptionists is that they may have domain-specific knowledge, suchas knowledge of medicine and medical terminology, which enables them tointerpret ambiguities in speech and thereby to improve transcriptaccuracy. Human transcriptionists, however, have a variety ofdisadvantages. For example, human transcriptionists produce transcriptsrelatively slowly and are subject to decreasing accuracy over time as aresult of fatigue.

Various automated speech recognition systems exist for recognizing humanspeech generally and for transcribing speech in particular. Speechrecognition systems which create transcripts are referred to herein as“automated transcription systems” or “automated dictation systems.”Off-the-shelf dictation software, for example, may be used by personalcomputer users to dictate documents in a word processor as analternative to typing such documents using a keyboard.

Automated transcription systems, and speech recognizers more generally,use both “acoustic models” and “language models” to recognize speech. Ingeneral, an acoustic model maps audio signals to phonemes or parts ofphonemes. A phoneme is the smallest phonetic unit in a language that iscapable of conveying a distinction in meaning, such as the “m” in “mat”and the “b” in “bat.” During speech recognition, an acoustic model isused to identify the phonemes represented by portions of the audiosignal being recognized. Such a sequence of phonemes may then becombined to recognize the words, sentences, and other syntactic elementsspoken by the speaker. Various kinds of acoustic models, such as thosewhich utilize Hidden Markov Models (HMMs), are well-known to thosehaving ordinary skill in the art.

A particular acoustic model represents a particular mapping betweenspeech and text. Although such a mapping could be specified manually bythe designer of the transcription system, manual creation of such amapping would be prohibitively time-consuming and would not likelyproduce an accurate acoustic model. Instead, acoustic models typicallyare created using a semi-automated process referred to as “training.”The term “training” refers to the process of adapting the parameters ofan acoustic model (or of a speech recognition system more generally) foroptimal performance in a new domain (e.g., medical or legal) and/or inconjunction with a new speaker.

Referring to FIG. 1A, a dataflow diagram is shown of a prior art system100 for training a set of acoustic models 112. In the system 100, theacoustic models 112 are trained using a training database 101 consistingof two closely connected data sources: (1) training speech 102 (e.g., inthe form of audio recordings of speech) in a particular target domainand/or from a particular speaker; and (2) verbatim transcripts 104 ofthe speech 102. Because the transcripts 104 are known to be verbatimtranscripts of the training speech 102, the combination of the trainingspeech 102 and transcripts 104 implicitly define mappings betweenphonemes and text, as required by acoustic models. The process oftraining may be viewed as a process by which these mappings areextracted from the training speech 102 and corresponding transcripts 104and then represented in a form which may be used subsequently to performspeech recognition on other speech 126 in the same domain. While“speaker dependent” systems can only reliably recognize speech spoken bythe speaker of the training speech 102, “speaker independent” systemsuse training speech spoken by several different speakers, andcorresponding transcripts, to train speaker-independent models which maybe used to recognize speech from any speaker.

More specifically, a dictionary 108 which maps text to phonetic symbolsis used to translate 106 the transcripts 104 into a sequence ofdictionary symbols 110 representing the sequence of phonemes in thetranscript 104. For example, the sentence “this is a cat” may betranslated into the following sequence of dictionary symbols: “dh ih sih s ax k ae t,” where each dictionary symbol represents a phoneme inthe original sentence.

A base set of acoustic models 112 may be predefined. Each of theacoustic models 112 typically is associated with a set of Gaussianmodels, each of which has a set of mean values and variances. Beforesuch models 112 have been trained, they may have initial values, such asmean values of zero and variances of some predetermined large number.From the acoustic models 112, a sequence of acoustic models 116corresponding to the dictionary symbols 110 may be identified 114. Morethan one acoustic model may correspond to each dictionary symbol.

An association is made between these models 116 and the training speech102 by aligning 118 the speech 102 onto the sequence of models 116,thereby producing timing data 120 specifying a temporal mapping betweenthe models 116 and frames in the training speech 102. A frame is a shortaudio segment, typically 5-10 milliseconds in duration. Each of theacoustic models 116 may be aligned with a plurality of frames. In theexample provided above, the “ih” models may be assigned to frames fromthe corresponding sound in speech for the word “this” as well as thesame sound in speech for the word “is.” Parameters of the models 116(such as their means and variances) may then be derived fromcharacteristics of the speech 102 in the corresponding frames. Suchderivation of acoustic model parameters, and subsequent updating of theacoustic models 112, is referred to as “training” 122 the acousticmodels 112. In general, the resulting parameter values indicateprobabilities that particular observed sounds represent particularphonemes or parts of phonemes.

The process just described may be repeated for multiple instances oftraining speech and corresponding verbatim transcripts. Once theacoustic models 112 have been trained in this manner, speech recognition124 may be performed on other speech 126 by using the trained acousticmodels 112 to identify the phonemes that most likely correspond toframes in the speech 126. Text 128 corresponding to the speech 126 maybe produced by reversing the mapping going from words to phonemes tomodels. Because the parameters of the acoustic models 112 were derivedfrom the correspondence between the training text 104 and the trainingspeech 102, speech recognition performed in this way will likely producepoor results if the training text 104 does not accurately represent thetraining speech 102.

As described above, acoustic models 112 typically are trained based on atraining database 101 which includes both recorded utterances 102 andtext transcriptions 104 which are known to be verbatim transcripts ofthe recorded utterances 102. In the dictation domain, for example, thedatabase 101 typically is created by first creating the text 104 andthen having speakers speak the text 104 to produce the training speech102. Text 104 typically is created or collected from existing sources.If a domain-specific acoustic model is desired, such existing sourcesmay be domain-specific sources, such as medical reports if amedical-specific acoustic model is desired. If a generic acoustic modelis desired, the existing sources may, for example, be text obtained froma newspaper.

Sections of the training text 104 may then be displayed to a speaker orspeakers, who may read the text aloud. A dedicated “speech collection”computer program may record the speech 102 and store it along with thecorresponding source text 104, thereby enabling a mapping between sourcetext 104 and spoken utterances 102 to be recorded.

In conversational systems, the training database 101 typically iscreated by manually transcribing either pre-existing speech or speechcreated specifically for the purpose of training. For example, chosensubjects may be asked to speak or converse on a given topic. Theresulting conversation may be recorded to produce training speech 102,and a human transcriptionist may listen to the spoken recording andproduce a verbatim transcript 104 of the speech 102. As a result, anaudio file, verbatim transcript of the audio file, and mapping betweenutterances in the audio file and words in the transcript 104 may beproduced.

Regardless of the manner in which the training database 101 is created,the quality of the resulting acoustic models 112 typically is highlyreliant on the accuracy of the correspondence between the trainingspeech 102 and the corresponding transcripts 104. In particular, it istypically required that there be an exact or near-exact temporalalignment between the training speech 102 and the correspondingtranscripts 104. If such a close temporal alignment does not exist, thenthe timing data 120 will specify a correlation between text (in thetranscripts 104) and audio (in the training speech 102) which do notrepresent the same speech as each other, and the resulting acousticmodels 112 will be poorly trained. Although some training systems areable to identify poorly trained phonemes and to discard the resultingtraining data (i.e., acoustic model parameters) in response, such anapproach reduces the amount of training data, which in turn reduces theaccuracy of the resulting acoustic models 112. For these reasons,verbatim transcripts typically are required for conventional acousticmodel training to be performed effectively.

It can be difficult to use such training techniques, therefore, indomains in which it is difficult to obtain a large quantity of trainingspeech and corresponding verbatim transcripts. Examples of such domainsinclude the medical and legal domains. In the case of the “promptedspeech collection” approach, it may be prohibitively expensive orotherwise impossible to enlist doctors, lawyers, and other professionalswho are able to spend the time necessary to recite large amounts oftraining text 104, and thereby to create the audio recordings 102necessary to produce the training database 101. Similarly, in the caseof the “conversational” approach, the abundance of obscuredomain-specific terms in the training speech 102 and the lack of trainedmedical/legal transcriptionists with knowledge of such terms may make itdifficult to produce the large volume of accurate verbatim transcripts104 that is needed for high-quality training to be performed. In eithercase, it may be difficult and/or prohibitively expensive to generate thetraining database 101, given the need for verbatim transcripts 104 oftraining speech 102 to perform conventional acoustic model training.

In some circumstances, however, large existing bodies of recorded speechand corresponding transcripts may exist. The medical transcriptionindustry, for example, regularly produces a variety of medical reportsbased on the recorded speech of doctors and other medical professionals.Such reports, however, typically are not suitable for use in the kind ofconventional acoustic model training illustrated in FIG. 1A, becausesuch reports typically are not verbatim transcripts of the recordedspeech for a variety of reasons.

One reason for a mismatch between the recorded speech and correspondingdocument is a failure by the transcriptionist to recognize andtranscribe the speech accurately. In addition to such errors, however,transcriptionists may intentionally introduce a variety of changes intothe written transcription. A transcriptionist may, for example, filterout spontaneous speech effects (e.g., pause fillers, hesitations, andfalse starts), discard irrelevant remarks and comments, convert datainto a standard format, insert headings or other explanatory materials,or change the sequence of the speech to fit the structure of a writtenreport as required by a certain medical institution or physician.

For example, referring to FIG. 12, an example of a structured andformatted medical report 1200 is shown. The report includes a variety ofsections 1202-1230 which appear in a predetermined sequence when thereport 1200 is displayed. In the particular example shown in FIG. 12,the report includes a header section 1202, a subjective section 1212, anobjective section 1224, an assessment section 1226, and a plan section1228. Sections may include text as well as sub-sections. For example,the header section 1202 includes a hospital name section 1210(containing the text “General Hospital”), a patient name section 1204(containing the text “Jane Doe”), a chart number section 1206(containing the text “851D”), and a report date section 1208 (containingtext “10/1/1993”).

Similarly, the subjective section includes various subjectiveinformation about the patient, included both in text and in a medicalhistory section 1214, a medications section 1216, an allergies section1218, a family history section 1220, a social history section 1222, anda signature section 1230. The objective section 1224 includes variousobjective information about the patient, such as her weight and bloodpressure. Although not illustrated in FIG. 12, the information in theobjective section may include sub-sections for containing theillustrated information. The assessment section 1226 includes a textualassessment of the patient's condition, and the plan subsection 1228includes a textual description of a plan of treatment. Finally, thesignature section includes a textual representation of the doctor'ssignature.

Note that information may appear in a different form in the report fromthe form in which such information was spoken by the dictating doctor.For example, the date in the report date section 1208 may have beenspoken as “october first nineteen ninety three, “the first of octoberninety three,” or in some other form. These alternative ways of speakingthe same date are referred to herein as “alternative spoken forms” ofthe date. More generally, each way of speaking a particular concept isreferred to herein as a “spoken form” of the concept. Thetranscriptionist, however, transcribed such speech using the text“10/1/1993” in the report date section 1208, perhaps because writtenreports in the hospital specified in the hospital section 1210 requiresthat dates be expressed in reports in such a format.

Similarly, information in the medical report 1200 may not appear in thesame sequence in the report 1200 as in the original audio recording, dueto the need to conform to a required report format or some other reason.For example, the dictating physician may have dictated the objectivesection 1224 first, followed by the subjective section 1212, and then bythe header 1202. The written report 1200, however, contains the header1202 first, followed by the subjective section 1212, and then theobjective section 1224. Such a report structure may, for example, berequired for medical reports in the hospital specified in the hospitalsection 1210.

The beginning of the report 1200 may have been generated based on aspoken audio stream such as the following: “this is doctor smith on uhthe first of october um nineteen ninety three patient ID eighty five oned um next is the patient's family history which i have reviewed . . . ”It should be apparent that a verbatim transcript of this speech would bedifficult to understand and would not be particularly useful.

Note, for example, that certain words, such as “next is a,” do notappear in the written report 1200. Similarly, pause-filling utterancessuch as “uh” do not appear in the written report 1200. Furthermore,certain terms, such as dates, have been recorded in the report 1200using particular canonical forms (e.g., in the report date section1208). In addition, the written report 1200 organizes the originalspeech into the predefined sections 1202-1230 by re-ordering the speech.As these examples illustrate, the written report 1200 is not a verbatimtranscript of the dictating physician's speech.

Although a report such as the report 1200 may be more desirable than averbatim transcript for a variety of reasons (e.g., because it organizesinformation in a way that facilitates understanding), the report is notuseful as training text in the traditional acoustic model trainingprocess described above with respect to FIG. 1A, precisely because thereport 1200 is not the kind of verbatim transcript required fortraditional acoustic model training.

In summary, although a large body of existing documents corresponding tospeech may be available in certain circumstances, such documents may notbe verbatim transcripts of the corresponding speech. If conventionalacoustic model training were applied to such speech and correspondingdocuments, the resulting acoustic models would be sub-optimal, perhapsto such an extent that they would not be suitable for use in speechrecognition.

It would be advantageous, however, to be able to use such reports totrain acoustic models because of the abundance of existing reports indomains such as medicine and law. Although new, verbatim, transcriptscould be generated based on existing recorded spoken audio streams,generating large volumes of such transcripts would be tedious,time-consuming, and costly. Furthermore, it would inefficiently requiretwo transcripts to be generated for each recorded audio stream (oneverbatim transcript to be used for acoustic model training, and onenon-verbatim transcript to be used for traditional purposes).

Referring to FIG. 1B, a dataflow diagram is shown of a prior art system150 which attempts to solve the problem just described. The system 150includes spoken audio 152 and a corresponding non-literal transcript 154of the audio 152, produced by a transcriptionist 156. As described inmore detail below, the non-literal transcript 154 includes informationfrom the audio 152, but is not a literal (verbatim) transcript of theaudio 152. An attempt is made, either manually or automatically, toalign 158 the audio 152 with the non-literal transcript 154, therebyproducing timing data 160 specifying temporal correlations betweenportions of the audio 152 and text in the non-literal transcript 1.54.

The audio 152, timing data 160 and non-literal transcript 154 areprovided to a confidence filter 164, which measures the degree of “fit”between the frames and corresponding word models. If the fit for aparticular frame does not satisfy a confidence threshold, the confidencefilter 164 marks the frame as unusable. The confidence filter 164thereby produces a set of filtered labels 166 which identify the framesthat satisfied the confidence threshold. The audio 152, non-literaltranscript 154, and filtered labels 166 are provided to a trainer 162,which produces a set of trained acoustic models 168 based on theportions of the spoken audio stream 152 and non-literal transcript 154identified by the filtered labels 166.

One problem with the approach illustrated in FIG. 1B is that a largeamount of training data from the initial acoustic models 164 may bediscarded because so much of the non-literal transcript 154 fails tomatch the corresponding portions of the spoken audio 152. In particular,such an approach may tend to systematically discard training data thatdo not take the same form as the text in the non-literal transcript 154.For example, if the word “November” in the spoken audio stream 152 isaligned with the text “11” in the non-literal transcript 154,such-training data will be discarded even though “November” and “11”represent the same semantic content. If the spoken audio stream 152consistently contains the word “November” when the non-literaltranscript contains the text “11”, training data of this kind willconsistently be discarded. The approach illustrated in FIG. 1B,therefore, has limited usefulness.

What is needed, therefore, are improved techniques for training speechrecognition systems and, in particular, improved techniques for trainingtranscription systems based on non-literal transcripts of speech.

SUMMARY

In one embodiment of the present invention, a system is provided fortraining an acoustic model for use in speech recognition. In particular,such a system may be used to perform training based on a spoken audiostream and a non-literal transcript of the spoken audio stream. Such asystem may identify text in the non-literal transcript which representsconcepts having multiple spoken forms. The system may attempt toidentify the actual spoken form in the audio stream which produced thecorresponding text in the non-literal transcript, and thereby produce arevised transcript which more accurately represents the spoken audiostream. The revised, and more accurate, transcript may be used to trainthe acoustic model, thereby producing a better acoustic model than thatwhich would be produced using conventional techniques, which performtraining based directly on the original non-literal transcript.

For example, in one aspect of the present invention, a method isprovided for use with a system including a first document containing atleast some information in common with a spoken audio stream. The methodincludes steps of: (A) identifying text in the first documentrepresenting a concept having a plurality of spoken forms; (B) replacingthe identified text with a context-free grammar specifying the pluralityof spoken forms of the concept to produce a second document; and (C)generating a first language model based on the second document.

In another aspect of the present invention, a method is provided whichincludes steps of: (A) generating a first language model based on afirst document including text and a context-free grammar specifying aplurality of spoken forms of a concept; and (B) using the first languagemodel in a speech recognition process to recognize a spoken audio streamand thereby to produce a transcript of the spoken audio stream.

In another aspect of the present invention, a method is provided whichincludes steps of: (A) applying a speech recognition process torecognize a spoken audio stream and thereby to produce a first documentusing a first language model based on a second document including textand a context-free grammar specifying a plurality of spoken forms of aconcept; and (B) using the first document and the audio stream to trainan acoustic model.

In another aspect of the present invention, a method is provided whichincludes steps of: (A) identifying a first document containing at leastsome information in common with a spoken audio stream; (B) identifyingtext in the first document representing a concept; (C) replacing theidentified text with a context-free grammar specifying a plurality ofspoken forms of the concept to produce a second document; and (D)replacing text in the second document with normalized text to produce anormalized document including samples of text belonging to spoken formsin the finite state grammar.

In another aspect of the present invention, a method is provided whichincludes steps of: (A) identifying a normalized document of a spokenaudio stream, the normalized document including a context-free grammarspecifying a plurality of spoken forms of a concept; (B) generating alanguage model based on the normalized document; (C) using the languagemodel in a speech recognition process to recognize the spoken audiostream and thereby to produce a second document; and (D) filtering textfrom the second document by reference to the normalized document toproduce a filtered document.

In another aspect of the present invention, a method is provided whichincludes steps of: (A) identifying a normalized document of a spokenaudio stream, the normalized document including a context-free grammarspecifying a plurality of spoken forms of a concept; (B) using thelanguage model in a speech recognition process to recognize the spokenaudio stream and thereby to produce a second document; and (C) using arobust parser to filter text from the second document by reference tothe normalized document to produce a filtered document.

Other features and advantages of various aspects and embodiments of thepresent invention will become apparent from the following descriptionand from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a dataflow diagram of a prior art system for training a setof acoustic models based on training speech and a corresponding verbatimtranscript;

FIG. 1B is a dataflow diagram of a prior art system for training a setof acoustic models based on training speech and a correspondingnon-literal transcript;

FIG. 2 is a flowchart of a method for training an acoustic model basedon a spoken audio stream and a non-literal transcript of the audiostream according to one embodiment of the present invention;

FIG. 3 is a dataflow diagram of a system for performing the method ofFIG. 2 according to one embodiment of the present invention;

FIG. 4 is a flowchart of a method that is used to replace text in thenon-literal transcript of FIG. 3 with a finite state grammar accordingto one embodiment of the present invention;

FIG. 5 is a dataflow diagram which illustrates techniques foridentifying a finite state grammar for use in the system of FIG. 3according to one embodiment of the present invention;

FIG. 6 is a flowchart of a method that is used in one embodiment of thepresent invention to generate the document-specific language model ofFIG. 3 according to one embodiment of the present invention;

FIG. 7 is a dataflow diagram of an alternative embodiment of a portionof the system of FIG. 3 which performs the method illustrated in FIG. 6according to one embodiment of the present invention;

FIG. 8 is a flowchart of a method that is used in one embodiment of thepresent invention to recognize the audio stream of FIG. 3 and thereby toproduce a document representing the audio stream;

FIG. 9 is a dataflow diagram of an alternative embodiment of a portionof the system of FIG. 3 which performs the method of FIG. 8 in oneembodiment of the present invention;

FIG. 10 is a flowchart of a method that is used in one embodiment of thepresent invention to train the acoustic model of FIG. 3;

FIG. 11 is a dataflow diagram of an alternative embodiment of a portionof the system of FIG. 3 which performs the method of FIG. 10 in oneembodiment of the present invention; and

FIG. 12 illustrates a textual medical report generated based on a spokenreport.

DETAILED DESCRIPTION

In one embodiment of the present invention, a system is provided fortraining an acoustic model for use in speech recognition. In particular,such a system may be used to perform training based on a spoken audiostream and a non-literal transcript of the spoken audio stream. Such asystem may identify text in the non-literal transcript which representsconcepts having multiple spoken forms. The system may attempt toidentify the actual spoken form in the audio stream which produced thecorresponding text in the non-literal transcript, and thereby produce arevised transcript which more accurately represents the spoken audiostream. The revised, and more accurate, transcript may be used to trainthe acoustic model, thereby producing a better acoustic model than thatwhich would be produced using conventional techniques, which performtraining based directly on the original non-literal transcript.

For example, referring to FIG. 3, a dataflow diagram is shown of asystem 300 according to one embodiment of the present invention fortraining a set of acoustic models 330 based on a spoken audio stream 302and a non-literal transcript 304 of the audio stream 302. The audiostream 302 may, for example, be a live or recorded spoken audio streamof a diagnosis and prognosis dictated by a doctor. The non-literaltranscript 304 may, for example, be a textual report (such as the report1200 shown in FIG. 12) generated based on the audio stream 302. Thenon-literal transcript 304 may, for example, be generated by a humantranscriptionist, an automated transcription system, or a combinationthereof.

Referring to FIG. 2, a flowchart is shown of a method 200 performed bythe system 300 of FIG. 3 according to one embodiment of the presentinvention. The non-literal transcript 304 is obtained (step 202), suchas by transcribing the audio stream 302. A concept identification unit306 identifies text 308 in the non-literal transcript representing aconcept having a plurality of spoken forms (step 204). Step 204 may berepeated multiple times to identify multiple instances of concept text308, each representing a concept having a plurality of spoken forms.

The term “concept” as used herein includes, for example, semanticconcepts (such as dates, times, numbers, codes, medications, medicalhistory, diagnoses, and prescriptions) and syntactic concepts (such asphrases, sentences, paragraphs, sections, and the full document). Aconcept may be spoken in a plurality of ways. Each way of speaking aparticular concept is referred to herein as a “spoken form” of theconcept. Therefore, any semantic or syntactic content having a pluralityof spoken forms is an example of a “concept” as that term is usedherein. For example, a speaker may indicate the end of a sentence bysaying “period”, “next sentence”, or “full stop”. Therefore, the end ofa sentence is an example of a “concept” as that term is used herein.

Consider, for example, the date October 1, 1993, which is a semanticconcept as that term is used herein. Spoken forms of this conceptinclude the spoken phrases, “october first nineteen ninety three,” “oneoctober ninety three,” and “ten dash one dash ninety three.” Text suchas “October 1, 1993” and “10/01/1993” are examples of “written forms” ofthis concept.

Now consider an example of a syntactic concept, such as the sentence“John Jones has pneumonia.” This sentence, which is a concept as thatterm is used herein, may be spoken in a plurality of ways, such as thespoken phrases, “john jones has pneumonia,” “patient jones diagnosispneumonia,” and “diagnosis pneumonia patient jones.” The writtensentence “John Jones has pneumonia” is an example of a “written form” ofthe same concept.

As yet another example, consider that there may be many ways to speak(or not to speak) a header for a particular section of a report. In themedical report 1200 (FIG. 12), for example, the section 1216 describingprevious medications may be preceded by spoken phrases such as,“Previous medications include . . . ”, “Prior meds are . . . ”, or“Patient previously prescribed . . . ” A human transcriptionist might,however, transcribe all of these alternative spoken forms into thesection heading, “Meds:”. When the transcriptionist encounters the priormedication information in the audio stream 302, the transcriptionist mayinsert such information into the previous medications section 1216 afterthe introductory text “Meds:”.

Now consider a document, such as the non-literal transcript 304, whichcontains a particular written form of a particular concept. Assume, forexample, that the transcript 304 contains the text “10/01/1993.” Thereis no way to know, a priori, which spoken form was spoken by the speakerof the audio stream 302 to produce the text “10/01/1993.” Although thespeaker may, for example, have spoken the phrase “ten dash oh one dashnineteen ninety three,” which may have been transcribed literally as“10/01/1993”, the speaker may alternatively have spoken the phrase“october first ninety three,” “first october ninety three,” or any otherspoken form of the same concept. The transcriptionist may have producedthe written form “10/01/1993” from any of the spoken forms because, forexample, such a written form is required by a particular written reportformat. The non-literal transcript 304, therefore, does not necessarilyinclude information about the spoken forms of concepts in the spokenaudio stream 302 which correspond to the written forms of the sameconcepts in the non-literal transcript 304.

As described above, however, it is desirable to obtain a verbatimtranscript of the spoken audio stream 302 for use in training theacoustic model 330. As further described above, however, it may beprohibitively difficult or expensive to generate such a verbatimtranscript from scratch. As will be described in more detail below, thisproblem is addressed in various embodiments of the present invention byusing alternative spoken forms of the concept(s) identified by theconcept identifier 306 as hints to a speech recognizer 322, whichrecognizes the spoken audio stream 302 and creates a improved transcript326 which is intended to more closely represent a verbatim transcript ofthe audio stream 302 than the original non-literal transcript 304. Theimproved transcript 326 may then be provided to a trainer 328 to producethe acoustic models 330, thereby improving the quality of the acousticmodels 330.

More specifically, in the embodiment illustrated in FIGS. 2 and 3, theconcept identifier 306 identifies text 308 representing a concept whichhas a plurality of spoken forms. The concept identifier 306 need not,however, identify the plurality of spoken forms. The concept identifier306 may identify the concept text 308 (step 204) in any of a variety ofways. The concept identifier 306 may, for example, be configured torecognize text formats (i.e., patterns) which are likely to representconcepts having multiple spoken forms. For example, the conceptidentifier 306 may be configured to identify text having the format“DD/MM/YYYY” (e.g., “01/10/1993”), “MM/DD/YYYY” (e.g., “10/01/1993”),“MMM DD, YYYY” (e.g., “Oct 01, 1993”), or “DD MMM YYYY” (e.g., “01 Oct1993”) as text representing a date. The concept identifier 306 maysimilarly be configured to recognize written forms which representalternative spoken forms of other kinds of concepts, such as numbers,diagnoses, and medications.

In the case of a syntactic concept such as the previous medicationssection 1214, the concept identifier 306 may be configured to recognizeany of a predetermined set of written forms (such as “Meds:” or“Medications”) as indicating the onset of the previous medicationssection 1214.

The concept identifier 306 may indicate the concept text 308 in any of avariety of ways. In general, the concept identifier 306 marks theconcept text 308 with a name or other unique identifier of thecorresponding concept. In one embodiment of the present invention, forexample, the concept identifier 306 inserts markup into the non-literaltranscript 304 which delimits the concept text 308. The non-literaltranscript 304 may, for example, be represented in the Extensible MarkupLanguage (XML), and the markup may be represented using XML tags. Forexample, the text “10/01/1993” may be marked up as follows:“<DATE>10/01/1993</DATE>”. The start tag “<DATE>” and corresponding endtag “</DATE>” delimit the date concept text “10/01/1993”. The text maybe further marked up, such as by marking up the month, day, and year, asin <DATE><MONTH>10</MONTH><DAY>01</DAY><YEAR>1993</YEAR></DATE>. As anexample of a syntactic concept, the text “FamHx: Reviewed.” may bemarked up as follows: “<FAM HISTORY>Reviewed.</FAM HISTORY>”.

Note that the use of a markup language such as XML, however, is merelyone example of a way in which the concept identifier 306 may indicatethe concept text 308, and does not constitute a limitation of thepresent invention.

The identified concept text 308 in the non-literal transcript 304 isreplaced with a finite state grammar 312 which specifies a plurality ofspoken forms of the concept, thereby producing a document 316 that isreferred to here as the “grammar version” of the transcript 304 (step206). In general, a finite state grammar specifies a plurality of spokenforms for a concept and associates probabilities with each of the spokenforms. For example, a finite state grammar for the date October 1, 1993,might include the spoken form “october first nineteen ninety three” witha probability of 0.7, the spoken form “ten one ninety three” with aprobability of 0.2, and the spoken form “first october ninety three”with a probability of 0.1. The probability associated with each spokenform is an estimated probability that the concept will be spoken in thatspoken form in a particular audio stream. A finite state grammar,therefore, is one kind of probabilistic language model. The term“probabilistic language model,” as used herein, refers to any languagemodel which assigns probabilities to sequences of spoken words. Examplesof techniques that may be used to generate finite state grammars inaccordance with embodiments of the present invention will be describedin more detail below.

To perform step 206, a grammar identifier 310 identifies the finitestate grammar 312 (referred to in FIG. 3 as a “concept grammar”), whichspecifies a plurality of spoken forms of the concept represented by theconcept text 308. Examples of techniques that may be used by the grammaridentifier 310 to identify the concept grammar 312 will be described inmore detail below.

A grammar replacer 314 replaces the concept text 308 in the non-literaltranscript 304 with the concept grammar 312, thereby producing thegrammar form 316 of the transcript 304. The grammar form 316 of thetranscript 304, therefore, may include both “flat” text (i.e., textwhich need not be represented as a finite state grammar) and finitestate grammars (e.g., the concept grammar 312). Note that step 206 maybe repeated for each of a plurality of concepts having correspondingconcept texts and concept grammars, in which case the grammar replacer314 may replace a plurality of concept texts 308 in the non-literaltranscript 304 with a plurality of corresponding concept grammars 312.

Note further that since concepts may range from low-level conceptsspanning a few words (such as a date concept) to high-level conceptsspanning a paragraph, section, or even the entire document, the grammarreplacer 314 may replace any amount of text with a correspondinggrammar, up to and including the entire document. In general, a grammarrepresenting an entire document may represent, for example, alternativesequences in which sections of the document may be spoken. Techniquesfor implementing such a global document grammar are described in moredetail, for example, in the above-referenced patent application entitled“Automated Extraction of Semantic Content and Generation of a StructuredDocument from Speech.” As further described in that patent application,the global document grammar may be hierarchical. For example, the globaldocument grammar may contain a root node having child nodes representingthe sections of the document. Each such child node may have furtherchild nodes, which may represent concepts (such as sub-sections ordates) that may appear within the document sections. Therefore it shouldbe appreciated that steps 204-206 may be implemented to recursivelyreplace text in the non-literal transcript 304 with grammars having astructure that corresponds to the structure of the global documentgrammar.

Consider again the simple example in which the non-literal transcript304 includes the text “when compared to previous film from <DATE>October1, 1993</DATE>”. Let the text “[GRAMMAR(DATE(10/1/1993))]” represent afinite state grammar for the date October 1, 1993. Such a finite stategrammar includes a plurality of spoken forms for that date andcorresponding probabilities. After step 206, the grammar form of thenon-literal transcript 304 may therefore be represented as “whencompared to previous film from [GRAMMAR(DATE(10/1/1993))]”. From thisexample it can be seen that the grammar form 316 of the non-literaltranscript 304 may include both flat text (e.g., “this is doctor smithon”) and a finite state grammar (i.e., GRAMMAR(DATE(10/1/1993))) or areference to such a grammar.

A language model generator 318 generates a language model 320 based onthe grammar version 316 of the transcript 304 (step 208). The languagemodel 320 is referred to herein as a “document-specific” language modelbecause it includes probabilities of word occurrences which reflect thefrequencies of such occurrences in the document 316. Thedocument-specific language model 320 may, for example, be a conventionaln-gram language model. Examples of techniques that may be used togenerate the document-specific language model 320 will be described inmore detail below.

A speech recognizer 322 uses the document-specific language model 320 torecognize the spoken audio stream 302 and thereby to produce an improvedtranscript 326 (step 210). For reasons which will be described below,the improved transcript 326 will typically be a more accurate transcriptof the spoken audio stream 302 than the non-literal transcript 304.

In general, a speech recognizer typically uses both a language model andan acoustic model to perform speech recognition. Referring again to FIG.3, the speech recognizer 322 may be a conventional speech recognizerwhich uses a base acoustic model 324 as an acoustic model and uses thedocument-specific language model 320 as a language model to recognizethe spoken audio stream 302 and thereby to produce the improvedtranscript 326. As will be described in more detail below, the speechrecognizer 322 may interpolate the document-specific language model 320with another language model to produce improved recognition results.

A trainer 328 trains the acoustic models 330 based on the improvedtranscript 326 and the spoken audio stream 302 using conventionaltraining techniques (step 212). Because the transcript 326 more closelyrepresents a verbatim transcript of the audio stream 302 than thenon-literal transcript 304, the quality of the acoustic model 330 ishigher than if the non-literal transcript 304 had been used to train theacoustic model 330. Experimental results have indicated thatimprovements in accuracy of 10-20% may be obtained using the techniquesdisclosed herein relative to a baseline in which training is performedusing conventional non-literal transcripts.

It was stated above that the grammar identifier 310 may identify conceptgrammar 312, which includes: (1) a plurality of spoken forms of theconcept represented by concept text 308, and (2) a plurality ofcorresponding probabilities. Examples of techniques will now bedescribed for identifying the concept grammar 312.

Referring to FIG. 4, a flowchart is shown of a method that is used toreplace the concept text 308 in the non-literal transcript 304 with theconcept grammar 312 (FIG. 2, step 206) in one embodiment of the presentinvention. The concept grammar 312 is identified (step 402). Asdescribed above, step 402 may be performed by the grammar identifier310. The concept text 308 in the non-literal transcript 304 is replacedwith the concept grammar 312 (step 404). As described above, step 404may be performed by the grammar replacer 314.

Referring to FIG. 5, a dataflow diagram is shown which illustrates thegrammar identifier 310 in more detail according to one embodiment of thepresent invention. The grammar identifier 310 includes a repository 522of finite state grammars 520 a-n. Each of the grammars 520 a-ncorresponds to a different concept. For example, spoken forms 516 a maybe alternative spoken forms for a “date” concept, while spoken forms 516n may be alternative spoken forms for a “section” concept.

The grammars 520 a-n include spoken forms 516 a-n for the correspondingconcepts. The spoken forms 516 a-n are paired with probabilities 518 aof occurrence of those spoken forms. Prior to performance of the methodillustrated in FIG. 4, the grammar repository 522 may be generated by afinite state grammar generator 514 as follows. A set of baseline(“seed”) grammars are generated manually and used to populate the finitestate grammar repository 522. For example, a grammar for the header tothe patient medical history section 1214 (FIG. 12) may include thespoken forms “clinical history is,” “previous medical history,” and“history is,” based on the system designer's knowledge or belief thatsuch spoken forms may be used to introduce the patient medical historysection 1214.

The grammar identifier 310 may include or otherwise have access to a setof audio recordings 508, such as audio recordings of other speech in thesame domain as the audio recording 302, and/or other speech by thespeaker whose speech is recorded in the audio recording 302. The grammaridentifier 310 also includes or otherwise has access to a set ofverbatim transcripts 512 of the audio recordings 508. The verbatimtranscripts 512 may be generated, for example, by transcribing 510 theaudio recordings 508 or by using the techniques described above withrespect to FIG. 3.

A grammar transcript 524 may be generated from the non-literaltranscript 304 based on the set of baseline grammars, using thetechniques described above with respect to FIG. 3 for generating thegrammar transcript 316. The verbatim transcripts 512 may then be parsedagainst the grammar transcript 524 using, for example, the filteringtechniques described below with respect to FIG. 11. The finite stategrammar generator 514 may use the results of such parsing to identifythe frequencies with which spoken forms in the baseline grammars appearin the verbatim transcripts. The finite state grammar generator 514 mayuse such frequencies as initial probabilities for each of the spokenforms in the grammars 520 a-n in the grammar repository 522.Furthermore, any mismatches between text in the verbatim transcripts 512and spoken forms in the grammar repository 522 may be flagged and usedto improve the grammar repository (such as by added new spoken forms tothe grammars 520 a-n), as described in more detail below with respect toFIG. 11.

For example, the finite state grammar generator 514 may ascertain that90% of the dates in the verbatim transcripts 512 appear in the form“MM/DD/YY” (e.g., “ten slash one slash ninety-three”) and that 10% ofthe dates appear in the form “MMM D YYYY” (e.g., “october one nineteenninety three”). The finite state grammar generator 514 may use theserelative frequencies as the probabilities 518 a, assuming that grammar520 a is a “date” grammar.

The grammar identifier 310 includes a grammar selector 502 whichidentifies the name of the concept tagged in the concept text 308, usesthe identified concept name to identify the corresponding grammar in thegrammar repository, extracts the identified grammar from the grammarrepository 522, and provides the extracted grammar as the conceptgrammar 312.

Note that although in the example illustrated in FIG. 5, a singleconcept grammar 312 is selected for a single instance of concept text308 representing a single concept, in practice the grammar identifier310 may analyze the entire non-literal transcript 304 and all instancesof concept text 308 within it in a single pass to identify thecorresponding grammars more efficiently.

It was stated above with respect to FIGS. 2 and 3 that the grammarversion 316 of transcript 304 is provided to the language modelgenerator 318. The language model generator 318 need not, however,generate the language model 320 based directly or solely on the grammarversion 316 of transcript 304. For example, in one embodiment of thepresent invention, the grammar version 316 of transcript 304 isnormalized before being provided to the language model generator 318.

Referring to FIG. 6, a flowchart is shown of a method that is used inone embodiment of the present invention to generate thedocument-specific language model 320 (FIG. 2, step 208) based on anormalized version of the grammar version 316 of transcript 304.Referring to FIG. 7, a dataflow diagram is shown of an alternativeembodiment of a portion of the system 300 (FIG. 3) which may perform themethod illustrated in FIG. 6.

In the embodiment illustrated in FIG. 7, the system 300 includes atokenizer and normalizer 704. The term “tokenization” refers tosegmenting text into consistent words (tokens). The term “normalization”refers to replacing text with its canonical form. Various kinds oftokenization techniques are well-known to those having ordinary skill inthe art, such as splitting punctuation marks (e.g., periods and commas)from the words immediately preceding them. Various kinds ofnormalization are well-known to those having ordinary skill in the art,such as changing the case of all words to lowercase, changingpunctuation marks into textual representations their spoken forms (e.g.,changing “.” to “% period %”), and changing blank lines to the text “%new paragraph %”. For example, the words “the”, “The”, and “THE” may benormalized by converting all of them into the canonical form “the”. Ingeneral, normalization is used to convert an existing transcript into aform which more closely resembles a verbatim transcript of the samespeech.

Conventional normalization techniques have been applied to documentsconsisting of plain text. In embodiments of the present invention,however, the grammar version 316 of transcript 304 may include bothplain text and finite state grammars. In one embodiment of the presentinvention, normalization and creation of the grammar version 316 oftranscript 304 proceeds in multiple steps. First, the concept identifier306 marks up, or otherwise modifies, the non-literal transcript 304 toindicate any identified concepts, thereby producing a concept-markednon-literal transcript 702 (step 602). Plain text in the concept-markednon-literal transcript 702 is normalized by tokenizer/normalizer 704,such as by using conventional tokenization and normalization techniques(step 604). The resulting document 706 is referred to herein as anon-grammar normalized transcript for reasons that will become clearbased on the description below. Marked concepts in the concept-markedtranscript 702 remain unchanged in the non-grammar normalized transcript706.

The grammar replacer 314 and/or grammar identifier 310 replacesmarked-up concept text in the non-grammar normalized transcript 706 withcorresponding grammars to produce a normalized grammar transcript 710(step 606). The resulting normalized transcript 710, therefore, differsfrom a conventional normalized document in that it includes both plaintext and finite state grammars.

In one embodiment, a flat text generator 712 replaces each grammar inthe normalized grammar transcript 710 with all of its spoken forms,weighted by their associated probabilities, to produce a normalized texttranscript 714 which includes flat text and no finite state grammars(step 608). Alternatively, the flat text generator 712 may, for example,replace each grammar in the normalized grammar transcript 710 with itshighest-probability spoken form, or with a randomly-selected one of itsspoken forms.

The language model generator 318 then generates the document-specificlanguage model 320 based on the normalized text transcript 714 (step610), rather than based directly on the grammar version 316 oftranscript 304, as described above with respect to FIGS. 2-3. Techniquesfor generating a language model based on a grammar are well-known tothose of ordinary skill in the art. The resulting language model 320 maybe an n-gram language model having the same structure as any otherconventional n-gram language model, including class language models.

It was stated above with respect to the embodiments illustrated in FIGS.2 and 3 that the speech recognizer 322 recognizes the audio stream 302,and thereby produces the improved transcript 326, using the baseacoustic model 324 and the document-specific language model 320.Techniques that may be used by the speech recognizer 322 to generate theimproved transcript 326 according to various embodiments of the presentinvention will now be described in more detail.

Referring to FIG. 8, a flowchart is shown of a method that is used inone embodiment of the present invention to recognize the audio stream302 and thereby to generate the improved transcript 326 (FIG. 2, step210). Referring to FIG. 9, a dataflow diagram is shown of an alternativeembodiment of a portion of the system 300 (FIG. 3) which may perform themethod illustrated in FIG. 8.

The speech recognizer 322 identifies the base acoustic model 324 (step802). The base acoustic model 324 may be any acoustic model 324, such asan acoustic model generated based on the speech of the speaker who spokethe audio stream 302, or based on a variety of speakers speaking thesame language as the speaker of the audio stream 302. The base acousticmodel 324 may or may not be generated based on speech in the same domainas the audio stream 302.

In the embodiment illustrated in FIG. 9, the system 300 also includes abackground language model 902. The background language model 902 may,for example, be generated based on a large number of documents in thesame domain as the audio stream 302, whether or not such documents havecorresponding audio streams. Such documents may have the same form asthe normalized grammar transcript 710, i.e., they may include bothnormalized plain text and finite state grammars. The probabilities inthe background language model 902 may be based on the frequencies ofoccurrence of word sequences and of spoken forms in the finite stategrammars.

The background language model 902 may cover spoken audio data containingphrases not contained in the non-literal transcript 304 and whichtherefore are not covered by the document-specific language model 320.Therefore, in one embodiment of the present invention, the speechrecognizer 322 includes a language model interpolator 904 whichidentifies the background language model 902 (step 804) and thedocument-specific language model 320 (step 806), and interpolates thetwo language models 902 and 320 to produce an interpolated languagemodel 906 that has better coverage than either the background languagemodel 902 or the document-specific language model 320 standing alone(step 808).

The language model interpolator 904 may use any of a variety ofwell-known interpolation techniques to produce the interpolated languagemodel 906. In particular, the language model interpolator 904 may weightthe document-specific language model 320 more heavily than thebackground language model 902 in the interpolation process, usingwell-known techniques.

The speech recognizer 322 includes a speech recognition engine 908 whichperforms speech recognition on the audio stream 302, using the baseacoustic model 324 as its acoustic model and the interpolated languagemodel 906 as its language model, thereby producing the improvedtranscript 326 (step 810). The effect of using the interpolated languagemodel 906 in the speech recognition process is that the grammar version316 of transcript 304, reflected in the document-specific language model320, serves as an additional constraint on the background language model902 and thereby assists in improving the accuracy of the improvedtranscript 326.

Although the improved transcript 326 is expected to more closelyrepresent a verbatim transcript of the audio stream 302 than theoriginal non-literal transcript 304, the improved transcript 326 maycontain errors. Therefore, in one embodiment of the present invention,apparent errors in the improved transcript 326 are identified andremoved from the improved transcript 326 prior to using the improvedtranscript 326 for training. As a result, the quality of training may beimproved.

Referring to FIG. 10, a flowchart is shown of a method that is used inone embodiment of the present invention to train the acoustic model 330(step 212) using a filtered version of the improved transcript 326 toimprove the results of training. Referring to FIG. 11, a dataflowdiagram is shown of an alternative embodiment of a portion of the system300 (FIG. 3) which may perform the method illustrated in FIG. 10.

In the embodiment illustrated in FIG. 11, the system 300 includes afilter 1100. The filter 1100 identifies non-matching portions of theimproved transcript 326 and the normalized grammar transcript 710 (step1002), and indicates in the transcript 710 that the correspondingsections of the spoken audio stream 302 are unreliable and thereforeshould not be used for training. The resulting transcript 1102, in whichunreliable portions of the audio stream 302 are flagged, is referred toherein as a near-truth transcript 1102 (step 1004).

The filter 1100 may perform filtering (steps 1002 and 1004) in any of avariety of ways. For example, in one embodiment of the presentinvention, the filter 1100 is a robust parser which may be implementedusing any of a variety of techniques well-known to those having ordinaryskill in the art. A robust parser is capable of parsing a text (such asa recognized transcript) against a grammar to determine whether the textis consistent with the grammar. A robust parser is “robust” in the sensethat it is capable of identifying portions of the text and grammar asmatching each other even if the two portions do not align precisely witheach other. For example, a robust parser may compare the text “Previousmedications include” and “Previous medications of the patient include”and determine that the second sentence is the same as the first, withthe addition of the inserted text (“of the patient”). Robust parsers mayalso recognize matching text in two documents-despite other kinds ofdifferences, such as text deletions and substitutions.

When a robust parser detects a difference between a first document and asecond document, the parser may mark up the first document to indicatethe differences between it and the second document. In the exampleabove, a robust parser might mark up the first sentence as follows:“Previous medications <INSERT>of the patient</INSERT>include”. Therobust parser thereby indicates that the second sentence is the same asthe first sentence, with the exception of the specified inserted text.

The preceding discussion describes comparisons made by the filterbetween plain text in the improved transcript 326 and the normalizedgrammar transcript 710. Recall, however, that the normalized grammartranscript 710 may also include finite state grammars. When the robustparser encounters a grammar in the transcript 710, the parser mayattempt to match text in the improved transcript 326 with any of thealternative spoken forms in the grammar. If a match is found, the robustparser may treat the text in the improved transcript 326 as matching thegrammar in the transcript 710. If no match is found, the parser maytreat this as a mismatch in the same way as if two units of plain texthad mismatched.

In the embodiment illustrated in FIGS. 10 and 11, training is performedon the acoustic model 330 using the near-truth transcript 1102 and theaudio stream 302 (step 1006), rather than using the improved transcript326 directly, as illustrated in FIG. 3. The use of a robust parser mayenable a larger amount of the improved transcript 326 to be retained inthe near-truth transcript 1102 than if a simple character-by-characteror word-by-word comparison were employed. Because the quality oftraining increases with the amount of valid training data, the use of arobust parser may thereby increase training quality.

The process of filtering may also be used to discover additional spokenforms for concepts. Recall from FIG. 5 that the grammar identifier 310may include a plurality of grammars 520 a-n for a plurality of concepts.When filtering is performed, the filter 1100 may determine that althoughsome particular concept text in the improved transcript 326 does notmatch the corresponding text in the normalized grammar transcript 710,the corresponding text nonetheless contains elements of the conceptrepresented by the concept text and may therefore indicate an additionalspoken form of the concept. For example, assume that the improvedtranscript 326 contains the text “ten one ninety three” but that the“date” grammar (e.g., grammar 520 a) in the grammar repository 522 doesnot include a spoken form having the format “MM D YY”. The filter 1100may nonetheless recognize that the text “ten one ninety three” containsthe required elements for a date, namely three numbers representing amonth, a date, and a year, albeit not in the sequence specified by anyof the recognized spoken forms for a date. In response, the filter 1100may flag the text for further attention. More generally, the filter 1100may flag text for further attention if the same text repeatedly appearsin the normalized grammar transcript 710 instead of text that isexpected. For example, if the expected cue for the assessment section1226 is “Assessment,” but the recognized document 326 repeatedlycontains “Conclusions” as the cue for the assessment section 1226, thefilter 1100 may flag such text for further attention. The filter 1100may require that such non-matching text appear some minimum number oftimes in the recognized document 326 for such text to be flagged.

A human operator may take any of a variety of actions in response tosuch a flag. For example, the operator may determine that the flaggedtext represents a new spoken form for the corresponding concept, and inresponse create a new spoken form in the corresponding grammar based onthe written form of the text. For example, the operator may add the word“conclusions” as an additional spoken form in the “assessment sectioncue” grammar.

Various intermediate and final results of the processes described abovemay be fed back into subsequent iterations of the processes to improvetheir future performance. For example, the base acoustic model 324 maybe replaced with the trained acoustic model 330 in a subsequentiteration of the process, thereby improving the quality of the improvedtranscript 326 produced by the speech recognizer 322, which in turn mayimprove the quality of the trained acoustic model 330 produced in thesubsequent iteration. Similarly, any language model that is trained bythe trainer 328 may be used to replace the background language model902, which may improve the quality of subsequent training for the samereasons. At each iteration, the speech recognizer 322 may be appliedagainst a set of testbed audio recordings to produce recognition resultsand the quality of the results measured. Subsequent iterations may beperformed until the recognition quality converges to within a specifiedrange.

As described above, the filtering step may produce additional spokenforms, which may in turn improve the quality of the document-specificlanguage model, which in turn may improve the quality of the improvedtranscript 326, and the overall quality of the trained acoustic model330.

The filter 1100 may also be used to improve training results in otherways. For example, the output of the filter 1100 may be analyzed todetermine whether any particular words are consistently marked asmismatches by the filter 1100. The presence of a word that isconsistently marked as a mismatch by the filter 1100 may indicate thatthe word was recognized incorrectly by the speech recognizer 322. Thedictionary entry for such a word may be updated using a correspondingportion of the spoken audio stream 302, thereby improving subsequentattempts at recognizing the same word.

One advantage of embodiments of the present invention is that theyenable acoustic models and language models to be trained in the absenceof verbatim transcripts of speech. The ability to perform training usingnon-literal transcripts expands the range of documents that may be usedfor training, and thereby enables training quality to be improved.

For example, training techniques disclosed herein may perform trainingsuccessfully even if a non-literal transcript transcribes a conceptusing a written form that differs from the spoken form of the conceptfrom which the written form was derived. This is true both for semanticconcepts (such as dates, times, and diagnoses) and syntactic concepts(such as sentences, paragraphs, and sections). For example, the text“10/1/1993” may be trained against the spoken forms “october onenineteen ninety three,” and “one october ninety three,” and “tenth ofoctober ninety three.” This decreases the amount of training data thatis discarded, thereby increasing the quality of the resulting acousticmodels.

Domains, such as the medical and legal domains, in which there are largebodies of pre-existing recorded audio streams and correspondingnon-literal transcripts, may find particular benefit in techniquesdisclosed herein. Training may be performed using such pre-existingaudio streams and non-literal transcripts, thereby providinghigh-quality acoustic models without the cost of producing new spokenaudio and/or verbatim transcripts. In particular, the existence of alarge amount of pre-existing data in such domains makes it possible totrain high-quality speaker-specific acoustic models, a task whichtypically is difficult or impossible to perform in the absence of largequantities of speaker-specific data. Once such acoustic models aregenerated, appropriate speaker-specific acoustic models may be selectedfor use in recognizing the speech of individual speakers, therebyimproving speech recognition accuracy in comparison to recognitionperformed using speaker-independent acoustic models.

Similarly, speaker-dependent language models may be generated using thelarge of pre-existing data that often exists in various domains.Referring again to FIG. 9, the background language model 902, forexample, may be a speaker-dependent language model that is generatedbased on a large number of documents representing the speech of aparticular speaker. Alternatively, the background language model 902 maybe a speaker-independent language model, and an additionalspeaker-dependent language model may be generated. This additionalspeaker-dependent language model may then be interpolated with both the(speaker-independent) background language model 902 and thedocument-specific language model 320 to produce the interpolatedlanguage model 906. The use of speaker-dependent language modelstypically improves the accuracy of speech recognition when applied tospeech of the same speaker.

Furthermore, techniques disclosed herein may be applied within suchdomains without requiring any changes in the existing process by whichaudio is recorded and transcribed. In the medical domain, for example,doctors may continue to dictate medical reports in their current manner,and transcripts of such reports may be produced in the current manner.Any new audio recordings and corresponding transcripts produced in thisway may be used for training in the manner disclosed herein. Alternativetechniques requiring changes in workflow, such as techniques whichrequire speakers to enroll (by reading training text), require speakersto modify their manner of speaking (such as by always speakingparticular concepts using predetermined spoken forms), or requiretranscripts to be generated in a particular format, may be prohibitivelycostly to implement in domains such as the medical and legal domains.Such changes might, in fact, be inconsistent with institutional or legalrequirements related to report structure (such as those imposed byinsurance reporting requirements). The techniques disclosed herein, incontrast, allow the audio stream 302 and corresponding non-literaltranscript 304 to be generated in any manner and to have any form.

In particular, techniques disclosed herein may operate independently ofand subsequent to the process by which the original audio stream 302 andcorresponding non-literal transcript 304 were created. For example, thesystem 300 need not be used to create the non-literal transcript based304 on the audio stream 302, or vice versa. The non-literal transcript304 may, for example, have been generated by a human transcriptionistprior to implementation and use of the system 300 in a particularsetting. Techniques disclosed herein, therefore, are independent notonly of the structure and content of the audio stream 302 and transcript304, but also of the processes by which the audio stream 302 andtranscript 304 were created.

As described above, techniques disclosed herein may identify multiplealternative spoken forms of a concept. This ability is useful in severalways. For example, the ability to identify multiple spoken forms of aconcept enables the document-specific language model 320 to reflectmultiple spoken forms of concepts, which in turn enables the speechrecognizer 322 to recognize speech in the audio stream 302 accuratelyeven if the concepts in the audio stream 302 take different spoken formsthan the same concepts in the original non-literal transcript 304. Theresult is that the improved transcript 326 represents a more accuratetranscript of the spoken audio stream 302 than would be possible if thedocument-specific language model 320 did not reflect multiple spokenforms. This helps to solve the problem caused by lack of alignmentbetween the non-literal transcript 304 and the audio stream 302, becausethe improved transcript 326 is likely to be more closely aligned thanthe non-literal transcript 304 with the audio stream 302. This improvestraining to the extent that training requires alignment between thetraining audio and the training text.

In addition to identifying alternative spoken forms, techniquesdisclosed herein assign probabilities to the spoken forms. The assignedprobabilities may, for example, be based on the relative frequency ofoccurrence of the spoken forms in the non-literal transcript 304 or inother training text. Such probabilities allow the actual spoken forms inthe audio stream 302 to be identified more accurately, even if thespoken forms in the audio stream 302 do not match the spoken forms inthe non-literal transcript 304. Such increased recognition accuracyimproves the quality of the trained acoustic models 330 for all of thereasons described above.

The use of finite state grammars to implement concepts havingalternative spoken forms enables a wide range of concepts to berecognized. As described above, such concepts include both semanticconcepts and syntactic concepts. There is no limitation on the number ofspoken forms that may be recognized for a particular concept, or on thedegree of variation among spoken forms for a particular concept. As aresult, essentially any concept having any set of alternative spokenforms may be implemented using techniques disclosed herein.

Improved alignment enables less training data to be discarded byfiltering than in previous systems. Furthermore, recall that prior artsystems tend to systematically discard training data that do not takethe same form as the transcript text. The filtering techniques disclosedherein avoid this problem by enabling speech to be used in training evenif the spoken form of the speech deviates from the corresponding writtenform in the training text. In particular, the use of grammarsrepresenting multiple spoken forms enables speech having any of thoseforms to be used in training, thereby increasing the efficiency andquality of training compared to previous systems.

Furthermore, the filter 1100 performs filtering by comparing theimproved transcript 326 to the normalized grammar transcript 710, ratherthan by filtering out results in the improved transcript 326 based onrecognition confidence measures. Because the normalized grammartranscript 710 includes alternative spoken forms for concepts, suchalternative spoken forms may be used by the filter 1100 to match text inthe improved transcript 326 and thereby to avoid filtering out text fromthe improved transcript 326 simply because it appears in a differentspoken form than the corresponding text in the non-literal transcript304. This additional use of alternative spoken forms further improvesthe results of training. Note, however, that even if the filter 1100were to perform filtering using the conventional approach based solelyon confidence measures, the resulting near-truth transcript 1102 wouldstill likely be more accurate than the original non-literal transcript304 due to the use of alternative spoken forms in the speech recognitionprocess itself.

As described above with respect to FIGS. 8-9, the speech recognizer 322may use an interpolated language model 906 to perform speechrecognition. The use of an interpolated language model 906 may improvethe quality of the improved transcript 326, and thereby the quality ofthe acoustic model 330, in comparison to a system which used only thedocument-specific language model 320. More specifically, the backgroundlanguage model 902 provides a breadth of coverage which is not providedby the document-specific language model 320, while the document-specificlanguage model 320 provides more detailed information about the likelycontent of the audio stream 302. Interpolating these two language models902 and 320 produces a language model which has the advantageouscharacteristics of both.

When training is performed by the trainer (e.g., FIG. 10, step 1006),any regions in the improved transcript 326 which have been marked (e.g.,by the filter 1100) as not matching the transcript 710 may be ignored,because such regions likely represent text which does not match thecorresponding portion of the audio stream 302. Note, however, thatthrough the use of the finite state grammars in the normalized grammartranscript 710, text in the improved transcript 326 which represents adifferent spoken form than text in the original non-literal transcript304 need not be discarded, but rather may be used in training. Thisresult differs from that which would be obtained if conventionaltraining techniques were employed, because such techniques (as describedabove) typically discard training data resulting from attempting totrain text representing one spoken form against audio in a differentspoken form. The quality of training in the embodiments disclosed hereinis thereby improved.

Statistics of frequent mismatches between the improved transcript 326and the normalized grammar transcript 710 collected over a large numberof training documents can be used to identify spoken language effectsthat are poorly covered by the current spoken form grammars 522. Thosestatistics can be used to automatically or manually add new (not yetcovered) variations for existing concepts, or to identify the need forthe entirely new concepts. The same statistics can help to identifydictionary problems (such as missing canonical forms or a newpronunciation variant that is specific to a given speaker or speakersub-population), which are otherwise hard to find.

The techniques disclosed herein may be used advantageously inconjunction with the techniques disclosed in the above-referenced patentapplication entitled, “Automated Extraction of Semantic Content andGeneration of a Structured Document from Speech.” For example, using thetechniques disclosed herein to identify the correspondence betweenwritten concepts and their spoken forms assists in the process ofconverting text generated by a speech recognition engine into a writtenform suitable for creating a structured document using the techniquesdisclosed in the above-referenced patent application.

It is to be understood that although the invention has been describedabove in terms of particular embodiments, the foregoing embodiments areprovided as illustrative only, and do not limit or define the scope ofthe invention. Various other embodiments, including but not limited tothe following, are also within the scope of the claims. For example,elements and components described herein may be further divided intoadditional components or joined together to form fewer components forperforming the same functions.

Although the term “transcript” is used herein to characterize variousdocuments (such as the non-literal transcript 304 and the grammarversion 316 of the transcript 304), such documents need not be“transcripts” of the audio stream 302. In particular, such documentsneed not be produced based on the audio stream 302. Rather, in generalthe “transcripts” described herein may be any documents which representinformation contained in the audio stream 302. Such documents may,however, contain some information which is not contained in the audiostream 302, and the audio stream 302 may contain some information whichis not in such documents. Such documents may be generated before orafter the audio stream 302 is generated. Although such documents may begenerated based on the audio stream 302, such as through transcription,the audio stream 302 may be generated based on such documents, such asby an enrollment process.

The term “verbatim transcript of an audio stream” refers to a documentthat includes a word-for-word transcription of the audio stream. Theterm “non-literal transcript of an audio stream” refers to any documentwhich is not a verbatim transcript of the audio stream, but whichincludes at least some of the same information as the audio stream.

Although the spoken audio stream 302 is described above as a “recorded”audio stream, this is not a limitation of the present invention. Rather,the audio stream 302 may be any audio stream, such as a live audiostream received directly or indirectly (such as over a telephone or IPconnection), or an audio stream recorded on any medium and in anyformat.

In the examples above, a distinction may be made between “finite stategrammars” and “text.” It should be appreciated that text may berepresented as a finite state grammar, in which there is a single spokenform having a probability of one. Therefore, documents which aredescribed herein as including both text and grammars may be implementedsolely using grammars if desired. Furthermore, a finite state grammar ismerely one kind of context-free grammar, which is a kind of languagemodel that allows multiple alternative spoken forms of a concept to berepresented. Therefore, any description herein of techniques that areapplied to finite state grammars may be applied more generally to anyother kind of context-free grammar.

Although the examples above only refer to language model interpolationinvolving the interpolation of one background language model with onedocument-specific language model, the background language model mayinclude multiple language models, such as a general medicine model, aspecialty model (e.g., radiology), and a speaker-specific model. It isalso possible to select a set of documents that are similar to orotherwise related to the non-literal transcript 304, and to build adocument-related background language model based on this set of relateddocuments.

In the embodiment described above with respect to FIGS. 6-7, the flattext generator 712 replaces grammars in the normalized grammartranscript 710 with flat text to produce the normalized text transcript714. Note, however, that this step is not a requirement of the presentinvention. For example, the flat text generator 712 and the normalizedtext transcript may be omitted. In other words, grammars in thenormalized grammar transcript 710 need not be replaced with flat text.Rather, the grammars may remain embedded in the normalized grammartranscript 710, and the document-specific language model 320 may begenerated based on the normalized grammar transcript 710, so long as thespeech recognizer 322 is capable of performing speech recognition basedon embedded grammars.

The techniques described above may be implemented, for example, inhardware, software, firmware, or any combination thereof. The techniquesdescribed above may be implemented in one or more computer programsexecuting on a programmable computer including a processor, a storagemedium readable by the processor (including, for example, volatile andnon-volatile memory and/or storage elements), at least one input device,and at least one output device. Program code may be applied to inputentered using the input device to perform the functions described and togenerate output. The output may be provided to one or more outputdevices.

Each computer program within the scope of the claims below may beimplemented in any programming language, such as assembly language,machine language, a high-level procedural programming language, or anobject-oriented programming language. The programming language may, forexample, be a compiled or interpreted programming language.

Each such computer program may be implemented in a computer programproduct tangibly embodied in a machine-readable storage device forexecution by a computer processor. Method steps of the invention may beperformed by a computer processor executing a program tangibly embodiedon a computer-readable medium to perform functions of the invention byoperating on input and generating output. Suitable processors include,by way of example, both general and special purpose microprocessors.Generally, the processor receives instructions and data from a read-onlymemory and/or a random access memory. Storage devices suitable fortangibly embodying computer program instructions include, for example,all forms of non-volatile memory, such as semiconductor memory devices,including EPROM, EEPROM, and flash memory devices; magnetic disks suchas internal hard disks and removable disks; magneto-optical disks;CD-ROMS; and DVDs. Furthermore, computer program instructions may betransmitted over any of a variety of network connections using any of avariety of network protocols, and executed during and/or aftertransmission. Any of the foregoing may be supplemented by, orincorporated in, specially-designed ASICs (application-specificintegrated circuits) or FPGAs (Field-Programmable Gate Arrays). Acomputer can generally also receive programs and data from a storagemedium such as an internal disk (not shown) or a removable disk. Theseelements will also be found in a conventional desktop or workstationcomputer as well as other computers suitable for executing computerprograms implementing the methods described herein, which may be used inconjunction with any digital print engine or marking engine, displaymonitor, or other raster output device capable of producing color orgray scale pixels on paper, film, display screen, or other outputmedium.

1. In a system including a first document, the document tangibly storedin a computer-readable medium and containing at least some informationin common with a spoken audio stream, a method performed by a computerprocessor executing instructions tangibly stored in a firstcomputer-readable medium, the method comprising steps of: (A)identifying text tangibly stored in the first document on a secondcomputer-readable medium, wherein the text represents a concept; (B)identifying, based on the identified text, a plurality of at least threespoken forms of the concept, including at least one spoken form notcontained in the first document, wherein all of the plurality of spokenforms have the same content as each other, wherein (B) comprises: (B)(1) identifying a name of the identified text; and (B) (2) using theidentified name to identify a corresponding context-free grammar in agrammar repository, wherein the corresponding context-free grammarspecifies the plurality of spoken forms of the concept; (C) replacingthe identified text with the corresponding context-free grammar toproduce a second document tangibly stored in a third computer-readablemedium; and (D) generating a first language model, tangibly stored in afourth computer-readable medium, based on the second document.
 2. Themethod of claim 1, wherein the concept comprises a semantic concept. 3.The method of claim 2, wherein the concept comprises a date.
 4. Themethod of claim 1, wherein the concept comprises a syntactic concept. 5.The method of claim 4, wherein the concept comprises a sentence.
 6. Themethod of claim 4, wherein the concept comprises the entire seconddocument.
 7. The method of claim 1, further comprising a step of: (E)using the first language model in a speech recognition process torecognize the spoken audio stream and thereby to produce a thirddocument tangibly stored in a fifth computer-readable medium.
 8. Themethod of claim 7, further comprising a step of: (F) using the thirddocument and the spoken audio stream to train an acoustic model tangiblystored in a sixth computer-readable medium.
 9. The method of claim 8,wherein the step (F) comprises steps of: (F)(1) filtering text from thethird document by reference to the second document to produce a filtereddocument tangibly stored in a seventh computer-readable medium; and(F)(2) using the filtered document and the spoken audio stream to trainthe acoustic model.
 10. The method of claim 9, wherein the step (F)(1)comprises applying a robust parser to the second and third documents toproduce the filtered document.
 11. The method of claim 7, wherein thestep (E) comprises steps of: (E) (1) interpolating the first languagemodel with a second language model to produce a third language modeltangibly stored in a sixth computer-readable medium; and (E) (2) usingthe third language model in the speech recognition process to recognizethe spoken audio stream and thereby to produce the third document. 12.The method of claim 1, wherein the first document comprises a documentgenerated based on the spoken audio stream.
 13. The method of claim 1,further comprising a step of: (E) prior to step (D), normalizing thesecond document to produce a normalized document tangibly stored in afifth computer-readable medium.
 14. The method of claim 1, furthercomprising a step of: (E) prior to step (D), repeating steps (A), (B),and (C) for each of a plurality of texts in the first document.
 15. Themethod of claim 1, wherein the step (C) comprises steps of: (C)(1)generating probabilities for the plurality of spoken forms specified bythe context-free grammar; and (C)(2) including the probabilities in thecontext-free grammar.
 16. The method of claim 15, wherein the step (C)further comprises a step of: (C)(3) including the plurality of spokenforms in the context-free grammar.
 17. The method of claim 1, whereinthe context-free grammar comprises a finite state grammar.
 18. Themethod of claim 1, wherein the first document comprises one of a firstplurality of documents tangibly stored in the second computer-readablemedium, and wherein the method further comprises a step of: (E)repeating steps (A), (B), and (C) for each of the plurality of documentsto produce a second plurality of documents, including the seconddocument, tangibly stored in a fifth computer-readable medium; andwherein the step (D) comprises a step of generating the first languagemodel based on the second plurality of documents.
 19. The method ofclaim 1, wherein all of the plurality of spoken forms have the samesemantic meaning as each other.
 20. A system comprising: a firstcomputer-readable medium tangibly storing a first document containing atleast some information in common with a spoken audio stream; a secondcomputer-readable medium tangibly storing computer program instructionsfor identifying text in the first document representing a concept; athird computer-readable medium tangibly storing computer programinstructions for identifying, based on the identified text, a pluralityof at least three spoken forms of the concept, including at least onespoken form not contained in the first document, wherein all of theplurality of spoken forms have the same content as each other, whereinidentifying the plurality of spoken forms of the concept comprises:identifying a name of the identified text; and using the identified nameto identify a corresponding context-free grammar in a grammarrepository, wherein the corresponding context-free grammar specifies theplurality of spoken forms of the concept; a fourth computer-readablemedium tangibly storing computer program instructions for replacing theidentified text with the corresponding context-free grammar to produce asecond document tangibly stored on a fifth computer-readable medium; anda sixth computer-readable medium tangibly storing computer programinstructions for generating a first language model based on the seconddocument.
 21. The system of claim 20, wherein the concept comprises asyntactic concept.
 22. The system of claim 20, further comprising: aseventh computer-readable medium tangibly storing computer programinstructions for using the first language model in a speech recognitionprocess to recognize the spoken audio stream and thereby to produce athird document tangibly stored in an eighth computer-readable medium.23. The system of claim 22, further comprising: a ninthcomputer-readable medium tangibly storing computer program instructionsfor using the third document and the spoken audio stream to train anacoustic model tangibly stored in a tenth computer-readable medium. 24.The system of claim 22, wherein the means for using the first languagemodel comprises: means for interpolating the first language model with asecond language model to produce a third language model tangibly storedin a ninth computer-readable medium; and means for using the thirdlanguage model in the speech recognition process to recognize the spokenaudio stream and thereby to produce the third document.
 25. The systemof claim 20, wherein the first document comprises a document generatedbased on the spoken audio stream.
 26. The system of claim 20, furthercomprising: means for normalizing the second document to produce anormalized document tangibly stored in a seventh computer-readablemedium.
 27. The system of claim 20, wherein the first document comprisesone of a first plurality of documents tangibly stored in the firstcomputer-readable medium, and wherein the system further comprises: aseventh computer-readable medium comprising tangibly storing computerprogram instructions for repeatedly activating the instructions foridentifying text, the instructions for identifying the plurality ofspoken forms, and the instructions for replacing the identified text foreach of the plurality of documents to produce a second plurality ofdocuments, tangibly stored in the fifth computer-readable medium,including the second document; and wherein the means for generating thefirst language model comprises means for generating the first languagemodel based on the second plurality of documents.
 28. The system ofclaim 20, wherein the concept comprises a semantic concept.
 29. Thesystem of claim 20, wherein all of the plurality of spoken forms havethe same semantic meaning as each other.
 30. A method performed by acomputer processor executing instructions tangibly stored in a firstcomputer-readable medium, the method comprising steps of: (A)identifying a first document, tangibly stored in a secondcomputer-readable medium, containing at least some information in commonwith a spoken audio stream; (B) (B)(1) identifying text in the firstdocument representing a concept; (B)(2) identifying a name of theidentified text; (B)(3) using the identified name to identify acorresponding context-free grammar in a grammar repository, wherein thecorresponding context-free grammar specifies a plurality of at leastthree spoken forms of the concept, and wherein the correspondingcontext-free grammar includes at least one spoken form not contained inthe first document, wherein all of the plurality of spoken forms havethe same content as each other; (C) replacing the identified text withthe corresponding context-free grammar to produce a second documenttangibly stored in a third computer-readable medium; and (D) replacingtext in the second document with normalized text to produce a normalizeddocument, tangibly stored in a fourth computer-readable medium, thenormalized document including samples of text belonging to spoken formsin the finite state grammar.
 31. The method of claim 30, wherein theconcept comprises a semantic concept.
 32. The method of claim 30,wherein the concept comprises a syntactic concept.
 33. The method ofclaim 30, wherein the context-free grammar comprises a finite stategrammar.
 34. The method of claim 30, wherein all of the plurality ofspoken forms have the same semantic meaning as each other.
 35. A devicecomprising: a first computer-readable medium tangibly storing computerprogram instructions for identifying a first document, tangibly storedin a second computer-readable medium, the first document containing atleast some information in common with a spoken audio stream; a thirdcomputer-readable medium tangibly storing computer program instructionsfor: (1) identifying text in the first document representing a concept;(2) identifying a name of the identified text; and (3) using theidentified name to identify a corresponding context-free grammar in agrammar repository, wherein the corresponding context-free grammarspecifies a plurality of at least three spoken forms of the concept, andwherein the corresponding context-free grammar includes at least onespoken form not contained in the first document, wherein all of theplurality of spoken forms have the same content as each other; a fourthcomputer-readable medium tangibly storing computer program instructionsfor replacing the identified text with the corresponding context-freegrammar to produce a second document tangibly stored in a fifthcomputer-readable medium; and a sixth computer-readable medium tangiblystoring computer program instructions for replacing text in the seconddocument with normalized text to produce a normalized document, tangiblystored in a seventh computer-readable medium, the normalized documentincluding samples of text belonging to spoken forms in the finite stategrammar.
 36. The device of claim 35, wherein the concept comprises asemantic concept.
 37. The device of claim 35, wherein the conceptcomprises a syntactic concept.
 38. The device of claim 35, wherein allof the plurality of spoken forms have the same semantic meaning as eachother.